1. Technical Field:
The present disclosure relates generally to an improved data processing system and method. More specifically, the present application is directed to a method and apparatus for efficient determination of memory copy versus registration in direct access environments.
2. Description of Related Art:
InfiniBand™ (IB) provides a hardware message passing mechanism which can be used for Input/Output (I/O) devices and Interprocess Communications (IPC) between general computing nodes. “InfiniBand” is a trademark and service mark of The InfiniBandSM Trade Association in the United States, other countries, or both. Consumers access InfiniBand™ message passing hardware by posting send/receive messages to send/receive work queues on an InfiniBand™ channel adapter (CA). The send/receive work queues (WQ) are assigned to a consumer as a queue pair (QP). Consumers retrieve the results of these messages from a completion queue (CQ) through InfiniBand™ send and receive work completions (WC). The source channel adapter takes care of segmenting outbound messages and sending them to the destination. The destination channel adapter takes care of reassembling inbound messages and placing them in the memory space designated by the destination's consumer.
There are two channel adapter types: host channel adapter and target channel adapter. The host channel adapter (HCA) is used by general purpose computing nodes to access the InfiniBand™ fabric. Consumers use InfiniBand™ verbs to access host channel adapter functions. The software that interprets verbs and directly accesses the channel adapter is known as the channel interface (CI).
Sending involves preparing a request work queue element (WQE) and posting it to the send queue of the queue pair. Receiving comprises posting a receive work queue element, which will be filled up when the data arrives. Completions, referred to here as completion queue entries (CQEs), are generated after these actions are done. The completion queue can be reaped by the user, or consumer, to find out the completion status of the I/O operation.
In order to avoid copying data across the application/kernel space boundary and through the kernel network stack, a user, such as an application in a host, may use a dedicated queue pair. In this case, the memory used in the send/receive queue pair is application memory. This requires that users have the ability to map and register memory with the channel adapter so that the adapter can use the user's memory buffers.
Generally, users map their buffers to enable direct access by the adapter to eliminate memory copies. This may lead to substantial gains in performance. Memory registration comes at a price, since the cost of memory registration is nontrivial. However, memory registration is very efficient when its cost is amortized over time when the consumer uses the same buffers and also in static environments.